Skip to content

Add anonymizer tool - #166

Open
sfraczek wants to merge 5 commits into
mainfrom
sfraczek/anonymizer
Open

Add anonymizer tool#166
sfraczek wants to merge 5 commits into
mainfrom
sfraczek/anonymizer

Conversation

@sfraczek

@sfraczek sfraczek commented Nov 24, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Introduces etradeAnonymizer — a Rust binary and anonymizer module for anonymizing E*TRADE / Morgan Stanley PDF statements while preserving data needed for tax calculations.

Subcommands

  • list <input.pdf> — lists all text tokens from FlateDecode/uncompressed streams (debugging/inspection).
  • anonymize <input.pdf> [output.pdf] — replaces PII with stable numeric tokens; output defaults to anonymous_<input>.pdf.

How it works

  • Operates directly on PDF streams (FlateDecode and uncompressed with an explicit /Length, PDF 1.3).
  • Preserves calculation-relevant text: strings containing CLIENT STATEMENT / For the Period, and everything between CASH FLOW ACTIVITY BY DATE and NET CREDITS/(DEBITS) (inclusive).
  • In-place replacement that keeps the exact original stream size (recompression + null-byte padding), so the PDF XREF table stays valid without full re-parsing.

Module layout

  • list.rs, replace.rs (engine: replace_pii_smart), path.rs, pdf.rs. No separate detect module/mode.

Testing

  • Unit tests run in CI (PDF token scanning, escape/unescape, output path handling).
  • 5 integration tests are marked #[ignore] — they require local plain PDF fixtures in anonymizer_data/ (git-ignored) and do not run in CI.

Notes

  • No openssl dependency, no bundled encrypted fixtures, no ANONYMIZER_TEST_KEY in CI.
  • Not a guarantee of complete PII removal — output should be reviewed manually before sharing.

@sfraczek
sfraczek requested a review from jczaja November 24, 2025 22:24
@sfraczek sfraczek added the enhancement New feature or request label Nov 24, 2025
@sfraczek
sfraczek marked this pull request as draft November 24, 2025 22:25
jczaja
jczaja reviewed Dec 8, 2025
View reviewed changes
Comment thread src/anonymizer/anonymizer.rs Outdated
@sfraczek
sfraczek force-pushed the sfraczek/anonymizer branch 2 times, most recently from c7fdef8 to 816af4d Compare December 13, 2025 18:12
@sfraczek
sfraczek force-pushed the sfraczek/anonymizer branch 2 times, most recently from ff026ea to 296708b Compare July 7, 2026 17:50
@sfraczek
sfraczek marked this pull request as ready for review July 7, 2026 17:50
@sfraczek
sfraczek requested a review from Copilot July 7, 2026 17:50
Copilot AI reviewed Jul 7, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new Rust-based “etradeAnonymizer” tool within the existing crate, aimed at detecting and replacing PII in (a constrained subset of) PDF statement streams while preserving original stream byte lengths to avoid rebuilding XREF offsets.

Changes:

  • Adds a new etradeAnonymizer binary with list and anonymize subcommands.
  • Implements PDF stream scanning/text token extraction plus “smart” token-preserving replacement with size-fitting recompression and padding.
  • Updates project metadata and dependencies to support the new functionality (e.g., flate2, sha2) and ignores local PDF fixtures.

Reviewed changes

Copilot reviewed 10 out of 12 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
src/lib.rs Exports the new anonymizer module from the crate root for library/tests/binary access.
src/anonymizer/mod.rs Defines the anonymizer module structure and re-exports submodules.
src/anonymizer/anonymizer.rs Adds the new CLI binary entry point and subcommand routing.
src/anonymizer/pdf.rs Implements strict PDF header validation, stream scanning, and token extraction/unescaping utilities.
src/anonymizer/replace.rs Implements smart anonymization by token replacement with recompression-to-fit and padding.
src/anonymizer/list.rs Adds a “list tokens” mode for inspecting extracted text tokens per stream.
src/anonymizer/path.rs Adds helper for generating default anonymized output filenames.
src/anonymizer/README.md Documents usage, strategy, limitations, and local-only testing workflow.
README.md Updates SPDX copyright year range.
Cargo.toml Registers the new binary and adds required dependencies.
Cargo.lock Updates the lockfile to include new/updated dependency versions.
.gitignore Ignores local PDF fixtures for offline integration testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/anonymizer/replace.rs
Comment thread src/anonymizer/replace.rs Outdated
Comment thread src/anonymizer/replace.rs Outdated
Comment thread src/anonymizer/replace.rs
Comment thread src/anonymizer/anonymizer.rs Outdated
Comment thread src/anonymizer/list.rs Outdated
Comment thread src/anonymizer/pdf.rs Outdated
Comment thread src/anonymizer/replace.rs
@sfraczek
sfraczek force-pushed the sfraczek/anonymizer branch 2 times, most recently from d45fb82 to c666272 Compare July 7, 2026 18:15
@sfraczek
sfraczek requested a review from Copilot July 9, 2026 22:16
Copilot AI reviewed Jul 9, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 12 changed files in this pull request and generated 3 comments.

Comment thread src/anonymizer/pdf.rs Outdated
Comment thread src/anonymizer/pdf.rs
Comment thread src/anonymizer/replace.rs Outdated
@sfraczek
sfraczek force-pushed the sfraczek/anonymizer branch from 4b9e581 to 5289612 Compare July 10, 2026 13:13
@sfraczek
Add build setup for the anonymizer tool
77f0024 
Register the etradeAnonymizer binary, add its dependencies (flate2, clap
derive), refresh the lockfile (incl. ethnum 1.5.3 for newer rustc), ignore
local PDF fixtures, and bump the SPDX copyright year.
@sfraczek
Add PDF parsing core for the anonymizer
9c796be 
Strict %PDF-1.3 reader, a single-regex stream scanner that classifies each
stream by its dictionary (font programs via /Length1 and unsupported filters
skipped, FlateDecode decoded, plain content scanned), byte-exact stream
slicing with end-marker validation, and a linear text-token scanner that
only commits tokens inside closed BT/ET text objects. Wires the module into
the crate root.
@sfraczek
Add smart anonymization engine
0aeef1d 
replace_pii_smart decompresses each content stream, replaces PII tokens with
positional indices while preserving marker phrases and the CASH FLOW section,
recompresses to fit, and NUL-pads to keep the original stream byte length so
the XREF table stays valid. Warns loudly on skipped streams and on a CASH
FLOW section that never closes.
@sfraczek
Add token listing, output-path helper, and CLI
0e5d52c 
The list subcommand prints extracted tokens per stream; anonymous_output_path
builds the default 'anonymous_<input>.pdf' name; the clap-based binary wires
the list and anonymize subcommands.
@sfraczek
Add anonymizer module documentation
0ec86c8
@sfraczek
sfraczek force-pushed the sfraczek/anonymizer branch from 5289612 to 0ec86c8 Compare August 20, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jczaja jczaja jczaja left review comments
Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sfraczek @jczaja